Skip to content

Always generate fast-mode data for NFD and NFKD tries #7222

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 7 commits into from
Nov 14, 2025
Merged

Always generate fast-mode data for NFD and NFKD tries #7222

merged 7 commits into from
Nov 14, 2025

Conversation

@hsivonen
Copy link
Member

@hsivonen hsivonen commented Nov 6, 2025

The intent is to allow the normalizer to assume that these tries are fast in the baked data mode.

This changeset removes the ability to generate serde-loadable data with small-mode tries for NFD and NFKD. Per previous discussion, we can add the ability to generate small tries for these back if there are users of ICU4X who really need that mode for these tries.

Fixes #6836

@hsivonen
Always generate fast-mode data for NFD and NFKD tries
2dc2bb1 
The intent is to allow the normalizer to assume that these tries are
fast in the baked data mode.

This changeset removes the ability to generate serde-loadable data
with small-mode tries for NFD and NFKD. Per previous discussion, we
can add the ability to generate small tries for these back if there
are users of ICU4X who really need that mode for these tries.

Fixes unicode-org#6836
robertbastian
robertbastian reviewed Nov 6, 2025
View reviewed changes
provider/data/normalizer/fingerprints.csv
Copy link
Member

@robertbastian robertbastian Nov 6, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

observation: 23% and 18% size increase

@Manishearth
Copy link
Member

Manishearth commented Nov 6, 2025

If you have perf numbers could you include them in the PR?

Manishearth
Manishearth reviewed Nov 6, 2025
View reviewed changes
@hsivonen
Copy link
Member Author

hsivonen commented Nov 14, 2025

This changeset is not about what it does alone. But even alone, some things get better (trimming some other improved zh scenarios):

utf16_throughput_nfc_vi/read                                                                             
                        time:   [1.1329 µs 1.1360 µs 1.1393 µs]
                        thrpt:  [1.7976 Gelem/s 1.8028 Gelem/s 1.8078 Gelem/s]
                 change:
                        time:   [-28.153% -27.837% -27.521%] (p = 0.00 < 0.05)
                        thrpt:  [+37.971% +38.575% +39.184%]
                        Performance has improved.
utf16_throughput_nfc_vi/writing_to_nfc                                                                             
                        time:   [1.2010 µs 1.2027 µs 1.2047 µs]
                        thrpt:  [1.7001 Gelem/s 1.7028 Gelem/s 1.7053 Gelem/s]
                 change:
                        time:   [-26.902% -26.718% -26.549%] (p = 0.00 < 0.05)
                        thrpt:  [+36.145% +36.460% +36.803%]
                        Performance has improved.
utf16_throughput_nfc_zh/read                                                                             
                        time:   [2.3713 µs 2.3741 µs 2.3771 µs]
                        thrpt:  [861.55 Melem/s 862.66 Melem/s 863.68 Melem/s]
                 change:
                        time:   [-63.103% -63.042% -62.979%] (p = 0.00 < 0.05)
                        thrpt:  [+170.11% +170.58% +171.03%]
                        Performance has improved.
utf16_throughput_nfc_zh/writing_to_nfc                                                                             
                        time:   [2.4370 µs 2.4404 µs 2.4439 µs]
                        thrpt:  [838.01 Melem/s 839.21 Melem/s 840.38 Melem/s]
                 change:
                        time:   [-62.319% -62.265% -62.210%] (p = 0.00 < 0.05)
                        thrpt:  [+164.62% +165.01% +165.38%]
                        Performance has improved.

It makes more sense to look at this changeset together with RUSTFLAGS="--cfg icu4x_unstable_fast_trie_only", which is unblocked by this changeset (numbers relative to the base changeset, not the above numbers):

utf16_throughput_nfc_el/read                                                                             
                        time:   [1.8041 µs 1.8075 µs 1.8117 µs]
                        thrpt:  [1.1305 Gelem/s 1.1330 Gelem/s 1.1352 Gelem/s]
                 change:
                        time:   [-30.688% -30.378% -30.087%] (p = 0.00 < 0.05)
                        thrpt:  [+43.035% +43.632% +44.275%]
                        Performance has improved.
utf16_throughput_nfc_el/writing_to_nfc                                                                             
                        time:   [1.8275 µs 1.8284 µs 1.8293 µs]
                        thrpt:  [1.1196 Gelem/s 1.1201 Gelem/s 1.1207 Gelem/s]
                 change:
                        time:   [-33.768% -33.641% -33.537%] (p = 0.00 < 0.05)
                        thrpt:  [+50.461% +50.697% +50.983%]
                        Performance has improved.
utf16_throughput_nfc_fr/read                                                                             
                        time:   [864.48 ns 868.02 ns 871.05 ns]
                        thrpt:  [2.3512 Gelem/s 2.3594 Gelem/s 2.3691 Gelem/s]
                 change:
                        time:   [-8.8472% -8.5285% -8.1823%] (p = 0.00 < 0.05)
                        thrpt:  [+8.9115% +9.3237% +9.7059%]
                        Performance has improved.
utf16_throughput_nfc_fr/writing_to_nfc                                                                             
                        time:   [885.76 ns 886.88 ns 888.03 ns]
                        thrpt:  [2.3062 Gelem/s 2.3092 Gelem/s 2.3121 Gelem/s]
                 change:
                        time:   [-13.313% -12.985% -12.465%] (p = 0.00 < 0.05)
                        thrpt:  [+14.241% +14.923% +15.358%]
                        Performance has improved.
utf16_throughput_nfc_vi/read                                                                             
                        time:   [893.95 ns 897.14 ns 900.13 ns]
                        thrpt:  [2.2752 Gelem/s 2.2828 Gelem/s 2.2910 Gelem/s]
                 change:
                        time:   [-43.266% -43.064% -42.844%] (p = 0.00 < 0.05)
                        thrpt:  [+74.959% +75.637% +76.262%]
                        Performance has improved.
utf16_throughput_nfc_vi/writing_to_nfc                                                                             
                        time:   [982.30 ns 991.01 ns 999.77 ns]
                        thrpt:  [2.0485 Gelem/s 2.0666 Gelem/s 2.0849 Gelem/s]
                 change:
                        time:   [-39.671% -39.154% -38.658%] (p = 0.00 < 0.05)
                        thrpt:  [+63.020% +64.349% +65.757%]
                        Performance has improved.
utf16_throughput_nfc_zh/read                                                                             
                        time:   [1.9505 µs 1.9534 µs 1.9565 µs]
                        thrpt:  [1.0468 Gelem/s 1.0484 Gelem/s 1.0500 Gelem/s]
                 change:
                        time:   [-70.002% -69.945% -69.890%] (p = 0.00 < 0.05)
                        thrpt:  [+232.11% +232.73% +233.36%]
                        Performance has improved.
utf16_throughput_nfc_zh/writing_to_nfc                                                                             
                        time:   [1.8302 µs 1.8312 µs 1.8324 µs]
                        thrpt:  [1.1177 Gelem/s 1.1184 Gelem/s 1.1190 Gelem/s]
                 change:
                        time:   [-71.814% -71.727% -71.653%] (p = 0.00 < 0.05)
                        thrpt:  [+252.77% +253.69% +254.78%]
                        Performance has improved.

Furthermore, this changeset also unblocks further improvements planned as #6925.

@hsivonen hsivonen merged commit befef1c into unicode-org:main Nov 14, 2025
30 checks passed
@hsivonen hsivonen deleted the fastnormtries branch November 14, 2025 10:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@robertbastian robertbastian robertbastian approved these changes

@sffc sffc Awaiting requested review from sffc sffc is a code owner

@Manishearth Manishearth Awaiting requested review from Manishearth Manishearth is a code owner

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Ship fast tries in icu_normalizer_data by default

3 participants

@hsivonen @Manishearth @robertbastian